[Spark] Define and use DeltaTableV2.startTransaction helper method #2053
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ryan-johnson-databricks
force-pushed
the
frj-delta-table-v2-helpers
branch
from
20ed084
to
cb742f8
Compare
|
LGTM! |
jaceklaskowski
approved these changes
Sep 16, 2023
| startTransaction(Some(snapshot)) | ||
|
|
||
| /** | ||
| * Starts a transaction for this table, using Some provided snapshot, or a fresh snapshot if None |
There was a problem hiding this comment.
That was intentional... Some/None as in Option.
| * Starts a transaction for this table, using Some provided snapshot, or a fresh snapshot if None | ||
| * was provided. | ||
| */ | ||
| def startTransaction(snapshotOpt: Option[Snapshot] = None): OptimisticTransaction = { |
There was a problem hiding this comment.
s/snapshotOpt/snapshot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
As part of making Delta more catalog-aware (see #2052), we need to solve two basic problems:
TableIdentifieroverload ofDeltaLog.forTable, the catalog lookup is performed internally and immediately discarded after extracting the table's storage location from it. If the caller needed the catalog info, they are out of luck. Caller can avoid the problem by creating aDeltaTableV2instead, which already provides bothDeltaLogandCatalogTable. To support this use case, we define new helper methods that make this easy to do (especially for unit tests).DeltaTableV2, there's no convenient way to start a transaction from it, in order to pass along the catalog info. To make it easy for callers to do the right thing, we define new helper methods for starting transactions directly from theDeltaTableV2itself. When transactions eventually become aware of catalog info, these new helper methods will make a narrow waist that can be enhanced to pass along their catalog info.How was this patch tested?
Existing unit tests (most changes are anyway in test code).
Does this PR introduce any user-facing changes?
Internal APIs only.